Digital video recorder (DVR) systems are becoming increasingly popular with consumers. Digital video recorder systems use magnetic hard disk drives rather than magnetic cassette tapes to store video programs. For example, the ReplayTV™ recorder and the TiVO™ recorder record television programs in digital formats on hard disk drives using, for example, MPEG-2 compression. Also, some DVR systems may record on a readable/writable digital versatile disk (DVD) rather than a magnetic disk.
MPEG-2 compression is essential for storing a recorded television program. An uncompressed video program would require an enormous (and impractical) amount of storage space on a disk drive. Similarly, multimedia systems require an enormous amount of bandwidth to provide services such as video-on-demand, videoconferencing, and the like. However, the inherent limitations on network bandwidth are a primary inhibitor to the performance of such systems.
Therefore, compression and communication standards have been developed to overcome the bandwidth limitations of conventional communication networks. These standards define the compression of video and audio data and the delivery of control data in a single bit stream transmitted in a frequency band that would before only accommodate an analog program.
Moving Picture Experts Group (MPEG) is a family of audio and video compression standards. In the MPEG-2 standard, video compression is defined both within a given picture (i.e., spatial compression) and between pictures (i.e., temporal compression). Spatial compression is accomplished by conversion of a digital image from the time domain to the frequency domain by a discrete cosine transform, quantization, variable length coding, and Huffman coding. Temporal compression is accomplished via a process referred to as motion compensation in which a motion vector is used to describe the translation of picture elements between picture frames.
ISO 13818-1 is the transport layer portion of the MPEG-2 standard, and specifies: i) packetization of audio and video elementary bit streams into packetized elementary streams (PESs), and ii) combination of audio and video PESs into a single time division or packet multiplexed bit stream for transmission and subsequent de-multiplexing into multiple bit streams for decompression and display. The single time division or packet multiplexed bit stream is as shown from various conceptual perspectives in FIGS. 1 to 5.
FIG. 1 illustrates a conceptual block diagram (generally designated 100) of the packetization of audio and video elementary bit streams and packet multiplexing according to an exemplary embodiment of the prior art. Distinct elementary streams are formed in audio encoder 105, video encoder 110, auxiliary (or other data) source 115, and systems data source 120. Each of these elementary streams is packetized into a packetized elementary stream (PES). The packetized elementary streams of audio data, video data, and the packets of other data and systems data are packet multiplexed by a MPEG-2 multiplexer into an MPEG-2 system stream.
FIG. 2 illustrates a conceptual block diagram of an exemplary time-division or packet-multiplexed bit stream (generally designated 200) according to an exemplary embodiment of the prior art. Bit stream 200 comprises a transport packet stream, wherein each packet illustratively comprises packet header 205 and payload 210 (i.e., packet data bytes) and, according to FIG. 2, optional adaptation field 215. An MPEG-2 bit stream comprises two layers, namely, a system layer (also referred to as an outer layer, a control layer, or the like) and a compression layer (also referred to as an inner layer, a payload layer, a data layer, or the like).
The MPEG-2 system layer facilitates (i) multiplexing one or more programs made up of related audio and video bit streams into a single bit stream for transmission through a transmission medium, and (ii) de-multiplexing of the single bit stream into separate audio and video program bit streams for decompression while maintaining synchronization. The system layer defines data stream syntax for timing control and synchronization and interleaving of the video and audio bit streams. The system layer is capable of: i) video and audio synchronization, ii) stream multiplexing, iii) packet and stream identification, iv) error detection, v) buffer management, vi) random access and program insertion, vii) private data, viii) conditional access, and ix) interoperability with other networks, such as those using asynchronous transfer mode (ATM). The MPEG-2 compression layer comprises the coded video and audio data streams. The system layer provides control data for multiplexing and de-multiplexing interleaved compression layers and, in doing so, defines those functions necessary for combining the compressed data streams.
FIG. 3 illustrates a conceptual block diagram of an MPEG-2-compliant decoding system (generally designated 300) according to an exemplary embodiment of the prior art. The components of decoding system 300 are well known to the skilled in the art and are therefore introduced for illustrative purposes only. Discussion of the functionality of these components will therefore be limited.
Decoding system 300 receives bit stream 200 as an input to system decoder 305. System decoder 305 de-multiplexes the system layer data of bit stream 200 into the compressed audio layer, the compressed video layer, and control data. The exemplary compressed audio layer data and video layer data are transferred to audio data buffer 310a and video data buffer 310v, respectively. The audio layer data is subsequently processed in audio decoder control block 315a and audio decoder 320a. The video layer data is subsequently processed in video decoder control block 315v and video decoder 320v. Exemplary control data is shown as program clock recovery (PCR) data, enable data, and startup values.
The MPEG-2 system layer supports a plurality of functions, namely, i) packet multiplexing and de-multiplexing of multiplexed multiple bit streams, ii) synchronous display of multiple coded bit streams, iii) buffer management and control, iv) time recovery and identification, v) random access, vi) program insertion, vii) conditional access, and viii) error tracking.
The MPEG-2 standard specifies two types of layer coding, namely, a program stream (PS) layer coding for relatively loss-less environments (e.g., CD-ROMS, DVDs, etc) and transport stream (TS) layer coding for lossy environments (e.g., cable television, satellite television, or other broadcast environments). Referring back to FIG. 2, bit stream 200 is illustratively a transport stream (TS) consisting of a plurality of TS packets divided into a packet header, an optional adaptation field, and the associated packet data (or payload). By contrast FIG. 4 illustrates a conceptual block diagram of a PES (generally designated 400) according to an exemplary embodiment of the prior art.
Packetized elementary stream (PES) 400 comprises packet header 405, optional PES header 410, and associated packet data 415. Packet header 405 comprises packet start code prefix 420, stream identifier (ID) 425, and PES packet length indicator 430. In accord herewith, all of the fields after PES packet length indicator 430 are optional. PES header 410 includes a presentation time stamp (PTS) field, a decoding time stamp (DTS) field, an elementary stream clock reference (ESCR) field, a elementary stream (ES) rate field, a DSM trick mode field, a copy information field, a prior PES clock recovery field, an extension field, and stuffing bytes.
Packet start code prefix 420 provides packet synchronization. Stream ID 425 provides packet identification and payload identification. PTS/DTS flag fields 435 and PTS/DTS fields 440 provide presentation synchronization. Data transfer is provided through the packet/header length 445, payload 415, and stuffing fields 450. Scramble control field 455 facilitates payload de-scrambling.
FIG. 5 illustrates a conceptual block diagram of an alternate time-division or packet-multiplexed bit stream (generally designated 200) according to an exemplary embodiment of the prior art. Bit stream 200 comprises access units 500, PES packets 400, and a plurality of TS packets 505. Bit stream 200 illustrates a layering relationship among access units 500, PES packets 400, and TS packets 505.
The TS layer operates to combine programs made up of PES-coded data with one or more independent time bases into a single stream. In accord with MPEG-2, a specific program does not require a unique time base, but if it does have a unique time base, the time base is the same for all of the elements of that program.
The PES layer is an inner layer portion of the MPEG-2 time division or packet multiplexed stream upon which the transport or program streams are logically constructed. The PES layer provides stream specific operations and supports the following: i) a common base of conversion between program and transport streams, ii) time stamps for video and audio synchronization and associated timing, especially for associated audio and video packets making up a broadcast channel, presentation, or program (collectively hereafter Programs), and having a common time base, iii) stream identification for stream multiplexing and de-multiplexing, and iv) such services as scrambling, VCR functions, and private data.
FIG. 5 further illustrates that, in accord with MPEG-2, each video or audio elementary stream (ES) is PES-packetized before insertion into a transport stream (TS). Elementary streams are continuous and PES packets containing an ES are generally of fixed lengths. Typically, video PES packets are on the order of tens of thousands of bytes and audio PES packets are on the order of thousands of bytes. However, video PES packets can also be specified as of undefined length. ES data, that is, access units 500, are first encapsulated into PES packets, which are, in turn, inserted into TS packets.
A transport stream may contain one or more independent, individual programs, such as individual broadcast television programs, whereby each individual program may have its own time base, and each stream comprises an individual program having its own packet identification (PID). Each separate individual program has one or more elementary streams generally having a common time base. While not illustrated in the PRIOR ART figures, different transport streams may be combined into a single system TS.
At the transport layer, the transport sync byte provides packet synchronization. The PID field data provides packet identification, de-multiplexing and sequence integrity data. The PID field is operable to collect the packets of a stream and reconstruct the stream. Continuity counters and error indicators provide packet sequence integrity and error detection. The payload unit start indicator and adaptation control are used for payload synchronization, while the discontinuity indicator and program clock reference (PCR) fields are used for playback synchronization. The transport scramble control field facilitates payload de-scrambling. Private data transfer is accomplished through the private data flag and private data bytes. The data bytes are used for private payload data transfer, and the stuffing bytes are used to round out a packet.
A transport stream is a collection of transport stream packets linked by standard tables. These tables carry program specific information (PSI) and are built when a TS is created at the multiplexer. These tables completely define the content of the stream. Two of the tables of the TS are the program association table (PAT) and the program map table (PMT). The PAT operates as a table of contents for the TS that contains a unique identifier for the stream, a version number enabling dynamic changes of the PAT and the TS, and an association table of pairs of values. The pairs of values, PN, and PMT-PID, are the program number (PN) and the PID of the tables containing the program.
The PMT, on the other hand, describes all streams comprising a program. Each entry in the PMT is related to one program. The PMT provides a mapping between packets and programs, and contains a program number that identifies the program within the stream, a descriptor to carry private information about the program, the PID of the packets that contain the synchronization information, a number of pairs of values (e.g., stream type (ST), Data-PID) which, for each stream, specify the ST and the PID of the packets containing the data of that stream or program (Data-PID).
Collectively, these tables are used to process a particular program. At any point in time, each program has a unique PID in the PMT, which provides the PIDs for the selected program's audio, video, and control streams. The streams with the selected PIDs are extracted and delivered to the appropriate buffers and decoders for reconstruction and decoding.
Achieving and maintaining clock recovery and synchronization is a problem, especially with audio and video bit streams. In accord with the MPEG-2 standard, an end-to-end constant delay timing model digital image and audio data take the same amount of time to pass through the system from encoder to decoder. The system layer contains timing information that requires constant delay. The clock references are program clock reference (PCR) and the time stamps are the PTS and DTS.
Synchronization is accomplished using the program clock reference (PCR) data field in the TS adaptation field. PCR is typically a 42-bit field that is coded in two parts, a PCR base having a 33-bit value in units of 90 kHz, and a PCR extension having a 9-bit extension in units of 27 MHz, where 27 MHz is the system clock frequency. As a general rule, the first 33 bits of the first PCR received by the decoder initialize the counter in a clock generation, and subsequent PCR values are compared to clock values for fine adjustment. The difference between the PCR and the local clock can be used to drive a voltage-controlled oscillator, or a similar device or function, for example, to speed up or slow down the local clock.
Audio and video synchronization is typically accomplished through the presentation time stamp inserted in the PES header. The presentation time stamp is a 33-bit value in units of 90 kHz, where 90 kHz is the 27 MHZ system clock divided by 300. The presentation time stamp value indicates the time that the presentation unit should be presented to the user.
In digital video systems (e.g., MPEG-2 compliant digital video systems and HDTV compliant digital video systems), the transport layer is used to carry audio data, video data, and system data streams, in packets, to the individual decoders and system memory. The individual streams are multiplexed together with a set of buffer assumptions as defined in the MPEG-2 System Layer specification.
In MPEG transport and program streams, synchronization among multiple elementary streams is accomplished with presentation time stamps (PTS) in the program streams and transport streams. As noted above, time stamps are generally in units of 90 KHz, but the system clock reference (SCR), the program clock reference (PCR) and the optional elementary stream clock reference (ESCR) have extensions with a resolution of 27 MHz. Decoding of N elementary streams is synchronized by adjusting the decoding of streams to a common master time base rather than by adjusting the decoding of one stream to match that of another. The master time base may be one of the clocks of the N decoders, the clock of the data source, or it may be some external clock.
Each program in a transport stream, which may contain multiple programs, may have its own time base. The time bases of different programs within a transport stream may be different. Because presentation time stamps apply to the decoding of individual elementary streams, they reside in the PES packet layer of both the transport streams and the program streams. End-to-end synchronization occurs when encoders save time stamps at capture time, the time stamps propagate with associated coded data to decoders, and the decoders use those time stamps to schedule presentations.
As noted above, synchronization of a decoding system with a channel is achieved through the use of the system clock reference (SCR) in the program stream and by its analog, the program clock reference (PCR), in the transport stream. The SCR and PCR are time stamps encoding the timing of the bit stream itself, and are derived from the same time base used for the audio and video presentation time stamp values from the same program. Since each program may have its own time base, there are separate PCR fields for each program in a transport stream containing multiple programs. In some cases it may be possible for programs to share PCR fields. A program shall have one and only one PCR time base associated with it.
Unfortunately, in a digital video recorder with packetized elementary stream (PES) storage, the systems time clock (STC) and the program clock reference (PCR) are lost when a video program is stored to the disk drive. The program is played back at a later time (i.e., time-deferred viewing). Therefore, there is a need in the art for an improved digital video recorder that implements a new synchronization apparatus and algorithm for audio and video synchronization.